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ABSTRACT 

In this paper we describe a “Train the Tutor” programme (TtT) for developing the 
metacognitive skills, facilitator skills, and tutor skills of students in a problem based 
learning (PBL) context. The purpose of the programme was to train 2 nd and 3 rd year 
undergraduate students in psychology to become effective PBL tutors for “freshmen ” 
(1 st year psychology students). Based on the 3C3R concept of Hung (2006), various 
instructional problems have been designed and used in a 6 steps training programme. 
The programme has been evaluated both in a formative and summative approach 
through a quasi-experimental control group design with pre- and post-measurements 
before and after the training programme. The study was conducted as part of a 
curriculum re-design for promoting problem based learning in psychology courses for 
undergraduate students in a university of applied science. The results indicate the 
importance of metacognitive skills of the tutor for effectively facilitating the learning 
process in a PBL context. 
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TUTOR COMPETENCIES IN A PEL LEARNING ENVIRONMENT 

From our understanding, Problem-based Learning (PBL) is a group based learning approach, 
in which the learners engage themselves in research and problem solving activities in order to 
gain a deeper understanding of theoretical concepts and the practical relevance of the problem 
they want to solve. This learning process needs to be supported by tutors who monitor and 
“scaffold” the learning process through guidance, coaching and observation. They interfere 
and support the learners when these are stuck in the process or lose direction. 

PBL tutors require a specific skill set and attitudes related to teaching and learning (Barrows, 
1988; Smith & Cook, 2012). On the one hand, PBL tutors must stimulate the students to get 
involved in a collaborative learning process, on the other hand the tutor must ensure that the 
students articulate suitable learning objectives and follow a structured procedure while 
exploring the topic. This requires both excellent facilitator skills and metacognitive skills, i.e. 
the ability to observe and reflect the effectiveness of the learning process, the learning 
strategies applied, and the group dynamic within the tutorial group (Brown, 1978; Flavell, 
1979; Kayashima & Inaba, 2011). Metacognitive skills need to be distinguished from 
cognitive skills (Veenmann, Van Hout-Wolters & Afflerbach, 2006). Cognitive skills refer to 
a person’s declarative and procedural knowledge in a certain domain, while metacognitive 
skills refer to knowledge of problem solving strategies, the ability how to organize and 
structure learning activities, and the understanding and the application of appropriate and 
effective learning strategies (Brown & DeLoache, 1978; Veenman, 2005). Table 1 shows a 
list of relevant metacognitive skills for learning as described by Hattie (2009). 


Table 1 


Overview: Metacognitive skills in a learning context, definitions and examples (Hattie, 2009, p. 190) 


Metacognitive skill 

Definition 

Example 

Organizing and 
transforming learning 

Overt or covert rearrangement of 
instructional materials to improve learning 

Making an outline before writing a 
paper 

Asserting self¬ 
consequences of 
learning 

Student arrangement or imagination of 
rewards or punishment for success or 
failure 

Putting off pleasurable events until 
work is completed 

Using self-instruction 

Self-verbalizing the steps to complete a 
given task 

Verbalizing steps of calculation in 
solving a maths problem 

Using self-evaluation 

Setting standards and using them for self¬ 
judgment 

Checking work completion before 
handing in to the evaluator 

Goal-setting / planning 

Setting of educational goals or planning 
sub-goals 

Making a list of items to accomplish 
during studying a certain subject 


Planning for sequencing, timing, and 
completing activities related to those goals 
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Self-monitoring 

Observing and tracking one’s own 
performance and outcomes, often 
recording them 

Keeping records of study output 

Develop task strategies 

Analysing tasks and identifying specific, 
advantageous methods for learning 

Creating mnemonics to remember 
facts 

Imagery 

Creating or recalling vivid mental images 
to assist learning 

Imagining the emotional and 
behavioural consequences e. g. after 
having passed a difficult exam. 


Effective PBL tutors “scaffold” the learning process in a way that guides the students without 
patronizing. (Smith & Cook, 2012). Through stimulating, probing, questioning, paraphrasing 
and providing feedback, the tutor stimulates the elaboration of the problem and directs the 
learning process rather than presenting the right answers to the problem at hand. Thus, the 
challenge for the tutor is how to steer and to guide the learners without lecturing or providing 
the students with predefined schemes or answers to the problem. 

In addition to these more process oriented interventions, which focus on the way how the 
students’ discussion in the tutorial is led and how the learning content is reflected upon, the 
tutor also needs to make sure that the students understand the content and the context of the 
problem they tackle. 

The 3C3R model of Hung (2006) (Figure 1) provides a framework that depicts six elements 
of process and content/context orientation in an effective PBL tutorial. It describes three 
structural elements (content, context, and connection) and three process elements 
(researching, reasoning, and reflecting). 


Reflecting 


Researching 



Reasoning 


Figure 1.: 3C3R Framework for designing a problem space in a PBL learning environment (Hung, 2006) 


39 






M. Miihlfelder, T. Konermann, L.-M. Borchard 


JPBLHE: VOL. 3, No. 2, 2015 


Whereas “content” focuses on the scope and depth of the problem, “context” refers to the 
applicability to a specific field of practice, and “connection” represents the connection to 
other knowledge domains. 

Considering relevant competencies of the tutor on the background of this framework, he or 
she should not only focus on process oriented questions that evoke researching, reasoning 
and reflecting among the students but he/she also needs to make sure that the content , context , 
as well as the connections with previous knowledge and related concepts are observed. Some 
researchers indicate that especially students with little or no experience need tutors with both 
high content oriented skills as well as process oriented skills (Davis, Naim, Paine, Anderson 
& Oh, 1992; Dolmans, Gijselaer, Moust, De Grave, Wolfhagen & Van der Vleuten, 2002; 
Zumbach, 2011). Leary, Walker, Shelton & Fitt (2013) report in their recent meta-analysis of 
the relevance of tutor background, tutor training and student learning a meaningful and highly 
significant effect size (Hedge’s g = 0.27, z = 6.75, p < 0.01, n = 223) for content expertise of 
the tutor. However, PBL tutors with high expertise and content knowledge need to be aware 
of the danger to direct and constrain the learning too much, thus stalling the students’ self- 
regulated learning process (Silver & Wilkinson, 1991). Chng, Yew and Schmidt (2011) have 
investigated the effect of social congruence between tutor and students on achievement and 
learning. They suggest that the ability of the tutor to communicate informally with students 
and to create a positive learning climate that promotes a free flow exchange of ideas, has a 
greater impact on learning at each of the PBL phases as compared to the tutor’s subject-matter 
expertise and the ability to explain concepts in a way that is easily understood by students. 

As a consequence the rationale for using peer facilitation in PBL with advanced students as 
tutors was based on the idea that through peer learning in small tutorial groups the students 
should be challenged by socially congruent peers to deeply reason, reflect and research the 
topic (3R) while the content, the context and the connection with the curriculum was fixed 
and provided by the faculty resp. the curriculum. A more practical reason for using peer 
students as PBL tutors in this particular case was the lack of qualified teaching resources that 
were sufficiently familiar with PBL methods and concepts. Hence, there was a strong need for 
an efficient and effective way to provide training for prospective PBL tutors as part of the 
new PBL curriculum. 


Training Rationale and Design 

Based on the recognized importance of metacognitive skills, facilitator skills, and tutor skills 
for effectively “scaffolding” the learning process of students, a training programme for 
prospective PBL tutors has been designed and evaluated in this study. The training 
programme was part of a wider curriculum transformation process for undergraduate 
psychology courses in a university of applied sciences. Problem based learning should 
become an integral element of the new curriculum, and developing a sufficient number of 
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qualified and certified PBL tutors was one of the critical contributing factors to the overall 
goal. One central design principle of the training was to use PBL as a core element for the 
training process itself. This means, the tutors were challenged to deal with ill-structured 
problems as they often arise during the tutorial process, such as observing and understanding 
group dynamics, dealing with students who try to get the “right” answers to the problem, or 
tutorial groups who are struggling with the definition of suitable learning goals, etc. 

The underlying assumption here was that the PBL methods should be learned at best in a 
context that resembles the learning settings which the tutors should create later for their own 
students (Sockalingam & Schmidt, 2011). 

Considering the role of the PBL tutor described in the previous section and acknowledging 
the relevant literature about the competencies needed by tutors to be effective, three major 
skill domains for tutor effectiveness have been identified (see also Barrows, 1988; Bertola & 
Murphy, 1994; Walsh, 2005): 

1. Metacognitive skills , such as reflecting the current learning situation, understanding 
the impact of own behaviour on student learning, and knowing and applying a variety 
of learning strategies. 

2. Facilitator skills, such as structuring the tutorial, creating a positive learning 
atmosphere, and leading through questioning and probing. 

3. Tutor skills, such as stimulating the learning process, re-stating the learning objectives, 
re-phrasing relevant learning content, and stimulation the discussion and interaction in 
the tutorial group. 

Hence, the main objectives of the “Train the Tutor” (TtT) programme have been defined as 
follows: 

1. Develop metacognitive skills for facilitating collaborative learning processes based on 
PBL principles. 

2. Learn facilitator skills for structuring the tutorial session (visualizing, summarizing, 
time keeping). 

3. Learn how to use appropriate tutor skills in order to scaffold and stimulate the learning 
process in a tutorial group (elaborating, directing, integrating, and constructively 
interacting with each other). 


The full “TtT” (“Train the Tutor”) programme took four months altogether. It was divided 
into six modules (each of which took between 0.75 and 2 days) and time in-between for 
preparation, documentation and follow-up. The total time invest for the training participants 
was 150 hours (60 hours seminar s/workshops, 90 hours for self-study, preparation, follow 
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up). The programme was designed and facilitated by an experienced PBL practitioner and 
faculty member. Table 2 displays the structure and content of the training programme. 


Table 2 

Structure and content for the “Train the Tutor” (TtT) programme _ 

Module No. Duration Training Objectives Training Content 


1.5 days 
(15 hours) 


Understand fundamentals of 
problem based learning 
(PBL) 

Understand the role of the 
PBL tutor 

Understand and practice basic 
facilitator skills 


History, goals and concepts of problem based 
learning 

The role, the attitude, and the required 
competencies of PBL tutors 

Basic facilitator skills (e. g. questioning, 
paraphrasing, stimulating, providing feedback) 


Follow up and preparation for module 2 (4 weeks) 
(Time invest: approx. 5 hours / week) 


2 


1.5 days 
(15 hours) 


Understand how to deal with • 

ill-structured problems # 

Understand der 3C3R model 
and its application 


Characteristics of ill-structured problems 

Content oriented and process oriented 
interventions (3C3R) 


Follow up and design of a problem case for module 3 (4 weeks) 
(Time invest: approx. 5 hours / week) 




• Understand group dynamics 

• Stimulating the systematic elaboration of 



in tutorial groups 

problems 


1.5 days 

• Practice effective tutor 

• Directing the learning process 

3 

(15 hours) 

interventions 

• Stimulating the integration of knowledge 




• Stimulating interaction and individual 




accountability 


Follow up and preparation for module 4 (4 weeks) 
(Time invest: approx. 5 hours / week) 


1 day 
4 

(6 hours) 


Learning from observing a 
role model 


Observing an experienced PBL tutor in action 
(plus briefing/debriefing) 


Follow up and preparation for facilitating a PBL tutorial (4 weeks) 
(Time invest: approx. 5 hours / week) 


5 

1 day 

(6 hours) 

• Experience self-efficacy as a • 

tutor 

• Practice acquired skills from 
modules 1 to 4 

Facilitate a PBL tutorial (plus observation and 
feedback by peers and master trainers) 

Follow up and documentation (2 weeks) 

(Time invest: approx. 5 hours / week) 

6 

1/2 day 

(3 hours) 

• Common reflection of the • 

training process and outcome 

• 

Reflect metacognitive skills, facilitator skills, and 
tutor skills acquired through the training 

Certification as a PBL tutor 

Total: 150 hours in four months 


(60 hours seminars/workshops, 90 hours for self-study, preparation, follow up) 
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EVALUATION OF THE TUTOR TRAINING PROGRAMME 

The PBL tutor training programme has been evaluated both in a formative (during the training 
process) and summative way (at the end of the training process). The purpose of the formative 
evaluation was to modulate, test and adapt content, methods and process of the training 
procedure. In addition, the summative evaluation aimed at allowing for comparing the 
effectiveness of the training programme in comparison to another form of tutor instruction 
and a control group. 

The major research question addressed in this evaluation study was: 


To which extent can the PBL training for tutors support the development of metacognitive 
skills, facilitator skills, and tutor skills of the training participants, compared to other forms 
of instruction (control group 1) and no formed training or instruction (control group 2)? 


In order to investigate this, a quasi-experimental research setting with repeated measures has 
been designed (Factor A: training group vs. control groups 1 and 2; Factor B: pre-measure 
and post-measure vs. post measure only). Factor A varies the intensity of training and 
instruction (1: PBL tutor training, 2: instruction through reading a tutor manual and guide, 3: 
no formal training or instruction), whereas factor B controls the influence of the pre-test on 
the post-test (1: pre- and post-test, 2: post-test only). The resulting evaluation design with the 
sample size of each cell is represented in table 3. 

Table 3 

Research Design: Factor A (Training group, Control groups 1 and 2), Factor B (pre- and 
post-test vs. post-test only) 


Factor B 



Bi: pre-test 
and post-test 

B 2 : without pre-test 
(post-test only) 

A i: 

Training group 

Training group 
(Aj Bp 
(n= 21) 

Training group 
(Ai B 2 ) 

(n=17) 

A2: 

Control group 

1 

Reading the “McMaster PBL tutor 
guide” (Walsh, 2005) 

(A 2 B,) 

(h=19) 

Reading the “McMaster PBL tutor 
guide” (Walsh, 2005) 

(A 2 B 2 ) 

(n=21) 

A3: 

No formal training or instruction 

No formal training or instruction 

Control group 

(A 3 B,) 

(A 3 b 2 ) 

2 

(h=20) 

(h=20) 
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Participants 

119 individuals (undergraduate psychology students in their second and third year and master 
students in their first year) participated in the evaluation study. All participants had completed 
fundamental modules in psychology before at least with satisfactory marks. The participants 
in the training group had been selected based on academic credits, and personal 
interest/motivation for facilitating PBL tutorials immediately after completion of the six 
modules. The remaining participants were assigned to the control groups in order to be trained 
later. Figure 2 displays a flow diagram which describes how the participants were streamed to 
the various cells in the quasi-experimental research setup. 



Figure 2. Flow diagram for visualizing the streaming of participants to the training group, control group 1 and 
control group 2 (own source) 


Even though this controlled and selective allocation of participants to the training group and 
the control groups limits the internal validity of the research design, we decided not to push 
back participants who were interested in the training in favour of other students who were not 
available to be actively engaged in the fast deployment of the PBL curriculum. Later we will 
discuss the consequences of this decision regarding the validity and generalizability of the 
results. 
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The participants’ average age was 22.2 years (SD = 4.2). 28 (25%) subjects were male, 85 
(75%) were female, which is a usual gender distribution in psychology undergraduate courses 
in Germany. 43 (38%) of the sample participants had previous experience as a learning 
facilitator e.g. in junior school or as trainers in youth sports clubs. 52 (46%) explained their 
interest to be engaged as a PBL tutor in the psychology study programme for undergraduates 
immediately after completion of the training programme or later. 

Tutorials 

The tutorials were part of an undergraduate course (1 st year) in personality psychology. These 
tutorials (90 min.) were accompanied by lectures once per week (90 min.). Based on the 
content of the lectures the students were assigned to discuss a specific problem or case 
thereafter and to formulate some learning goals as a preparation for the next lecture. This case 
was designed in such a way that it stated an ill-structured problem and triggered the 
discussion of the students related to the relevant concepts introduced in the lecture. Each 
tutorial group consisted of 10 to 12 students and was facilitated by a PBL tutor. Attending the 
tutorial was not mandatory for the students, however highly recommended by the faculty. On 
average, each group had one tutorial per week. Four to five tutorials ran in parallel. 

Measures 

The measures combined different sources of information by utilizing self-report measures of 
the tutor, behavioural measures of tutor effectiveness, as well as student satisfaction 
measures. Through this multi-method approach, a broader investigation into the effectiveness 
of the tutor training on metacognition, behaviour, and tutor effectiveness should be achieved. 

Tutor Skills Self-report (Questionnaire) 

In order to create a reliable and valid measure for self-perceived tutor skills, a questionnaire 
(28 items) with four scales has been designed. Each scale consisted of 7 items (see annex 1). 

1) MCSL (Meta-cognitive skills related to guiding learning groups) (e.g. “I have a large 
variety of behavioural strategies how to steer group dynamics.”) 

2) MCSR (Meta-cognitive skills related to self-regulation) (e.g. “I have a clear mental 
model of how to plan, do, and check my actions and their behavioural and emotional 
effects.”) 

3) FAS (Facilitator skills) (e.g. “I can easily structure group discussions.”) 

4) TUT (Tutor skills) (e.g. “I can easily evaluate different levels of knowledge and 
subject matter understanding of students in a tutorial group.”) 

The psychometric analysis of the questionnaire revealed sufficient internal consistencies for 
all four scales (Cronbach’sa: .69 - .78). Each item had to be rated by the subjects on a 5 
points Likert scale (1: strongly disagree, 2: disagree, 3: neutral, 4: agree, 5: strongly agree). 
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The sum of the scales was used as a measure for the self-reported metacognitive skills related 
to guiding learning groups (MCSL), self-regulation (MCSR), facilitator skills (FAS) and tutor 
skills (TUT). 

TIP (Tutor Intervention Profile) 

The “Tutor Intervention Prof11e“ (TIP) is a behaviour observation method and manual 
developed at the University of Maastricht (The Netherlands) in order to evaluate tutor 
behaviour effectiveness (De Grave, Dolmans & Van der Vleuten, 1998, 1999). It has been 
tested for reliability and validity and has been used as a method for tutor assessment in many 
cases. TIP encompasses four behavioural dimensions of tutor competencies regarding 
learning process-oriented interventions: (1) Stimulating elaboration, (2) Directing the learning 
process, (3) Stimulating the integration of knowledge, and (4) Stimulating interaction and 
individual accountability of the students. 


Table 4 displays the four behavioural dimensions for tutor effectiveness of the TIP and shows 
two example items for each dimension. 

Table 4 


Dimensions of die Tutor Intervention Profile (TIP) (De Grave, Dolmans & Van der Vleuten, 1998) 


Dimension 

Example 

1. Stimulating elaboration (SE) 

• 

... stimulates a more in-depth brainstorm by, for 
example, asking questions, asking for clarification, and 
stimulating relations. 


• 

.. .stimulates the identification of gaps in students' prior 
knowledge. 

2. Directing the learning process 

• 

... stimulates generating learning issues with sufficient 

(DLP) 


depth and width. 


• 

... draws the attention of students to gaps in prior 
knowledge while generating learning issues. 

3. Stimulation the integration of 

• 

... stimulates the integration on new acquired 

knowledge (SI) 


knowledge with knowledge acquired with previous 
cases. 


• 

... stimulates the students to apply the knowledge 
gained during self-study to explain the phenomena 
described in the case. 

4. Stimulating interaction and 

• 

... stimulates students to make an inventory of the 

individual accountability 
(SIINDACC) 


learning resources consulted during self-study. 

• 

... stimulates students to report out in their own words 
rather than reading from notes or photocopies. 


14 PBL tutors, who had completed the training programme before, have been assessed 
through peers and trainers, who observed the interaction between the tutor and the students 
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during the tutorial on a five point scale (0: not effective, 1: fairly effective, 2: moderately 
effective, 3: effective 4: highly effective). 

Learner Satisfaction Measures (Questionnaire). 

As a third measurement, the students who participated in the PBL tutorials rated the 
effectiveness of the tutor at the end of the tutorial on three items (see annex 2): 

1. Satisfaction with the learning outcome (SLO). This measure indicates overall student 
satisfaction with the learning outcome of the tutorial, directly at the end of the tutorial. 

2. Satisfaction with the learning process (SLP). This measure indicates the satisfaction 
with the learning process (pace and structure). 

3. Satisfaction with the learning content (SLC). This measure indicates the satisfaction 
with the relevance, depth and width of the learning content. 

Each student rated his or her level of satisfaction at the end of the tutorial on a 5 points Likert 
scale (“0” representing total dissatisfaction, “4” representing maximum satisfaction). 

Procedure 

The training group (n=38) followed the training programme as described in table 3. Control 
group 1 (n=40) was instructed to read the PBL tutor guide of the McMaster University in 
Hamilton, Canada, which is available online (Walsch, 2005). The rationale behind this was to 
test whether the resources invested in designing and implementing the PBL tutor training 
programme was justified in comparison with less expensive and less time consuming methods 
for preparing and instructing novices as PBL tutors. Control group 2 (n=41) did not receive 
any instruction or training. All subjects were pulled from the same population of 
undergraduate psychology students. The training group was selected based on personal 
interest and academic credits (see “participants” section). Control groups 1 and 2 were 
compiled randomly. Half of the subjects completed the tutor skills self-report (questionnaire) 
before the start of the training programme and at the end. The other half completed the 
questionnaire at the end of the programme only. The aim of this procedure was to control if 
the pre-test had an effect on the post-test. Only those who actually completed at least steps 1 
to 4 of the training programme were eligible as PBL tutors. Out of these, 14 tutors have been 
evaluated by peers and master trainer through observation and assessment with the TIP (Tutor 
Intervention Profile) and student assessment (see “measures”). Academic achievements, 
earlier experience as tutors in secondary school or clubs and motivation to become actively 
engaged as a PBL were recorded as control variables. 
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Research Hypotheses 

The following hypotheses should be tested in this quasi-experimental study. 


Hypothesis 1: The PBL tutor training should have significant positive effects on the 

facilitator skills, tutor intervention skills, and metacognitive skills of the 
training participants compared to the control groups 1 and 2. 

Hypothesis 2: The pre-test should have no effect on the post-test results for self-reported 

tutor skills. 


Hypothesis 3: Metacognitive skills, facilitator skills and tutor skills should be positively 

correlated with student satisfaction measures. 


Hypothesis 4: Self-reported tutor skills (questionnaire data) should be positively 

correlated with effective tutor behaviour as measured by the TIP (Tutor 
Intervention Profile). 


RESULTS 

Table 5 shows the correlations for all measures: Tutor skills self-report (questionnaire), 
observation of tutor behaviour with tutor intervention profile (TIP), and student satisfaction at 
the end of the tutorial. 
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The self-reported measures for metacognitive skills (MCSR, MSCL), facilitator skills (FAS) 
and tutor skills (TUT) are highly correlated. There are also strong correlations between the 
self-report (questionnaire) and the TIP ratings (observer ratings). Moreover, high correlations 
between student satisfaction measures (SLO, SLP, SLC) and self-reported facilitator and tutor 
skills are high. However, this holds true only regarding satisfaction with the learning outcome 
(SLO), not so much for satisfaction with the learning process (SLP) and the learning content 
(SLC). 

The training group had higher scores on all four scales in the post-test compared to the pre¬ 
test (see Figure 3). The control groups had partly higher and lower scores. On the TUT scale 
there was even a drop between pre-test and post-test for control group 1. 



■n 

T i 

i i 


Treatment group 


| Pretest: Meta-cognitive skills related to guiding learning groups (MCSL) 
] Posttest: Meta-cognitive skills related to guiding learning groups (MCSL) 










Group 

| Pretest:_Meta-cognitive skills related to self regulation (MCSR) 

] Posttest: Meta-cognitive skills related to self regulation (MCSR) 


HI Pretest: Facilitator skills (FAS) 

1 I Posttest: Facilitator skills (FAS) 




H Pretest: Tutor skills (TUT) 
I 1 Posttest: Tutor skills (TUT) 


Figure 3. Tutor skills self-report fpre-test, post-test) (standardized scale values for MCSL (Metacognitive skills 
related to guiding learning groups) MCSR (Metacognitive skills related to self-regulation), FAS (Facilitator 
skills) and TUT (Tutor skills) 


Applying a MANOVA procedure with factors A (training vs. reading the tutor guide vs. no 
formal training and instruction) and B (pre- and post-test vs. post-test only) for the four 
dependent variables MCSL, MCSR, FAS, and TUT showed a highly significant effect for 
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factor A, no effect for factor B, and a highly significant interaction between factors A and B 
for the dependent variable MCSL (Table 6). 

Table 6 

Multivariate Analysis of Variance (MANOVA) - Statistics for dependent variables MCSL 
(Metacognitive skills related to guiding learning groups), MCSR (Metacognitive skills related to self- 
regulation), FAS (Facilitator skills), TUT {Tutor skills) _ 


Source of Variation 

Dependent 

Variables 

Sum of 
Squares 
(Type III) 

Df 

Mean of 
Squares 

F 

Sig. 

Partial 

Eta 2 

Adjusted model 

MCSL 

305.40“ 

5 

61.08 

5.75 

.000 ** 

.213“ 


MCSR 

160.91 b 

5 

32.18 

2.22 

.058 

.095 b 


FAS 

146.49“ 

5 

29.30 

2.81 

.020* 

.117“ 


TUT 

300.62“ 

5 

60.12 

4.48 

.001 ** 

.175“ 

Constant term c 

MCSL 

67807.27 

1 

67807.27 

6384.08 

.000 

.984 


MCSR 

81695.18 

1 

81695.18 

5631.16 

.000 

.982 


FAS 

78002.21 

1 

78002.21 

7466.72 

.000 

.986 


TUT 

63794.01 

1 

63794.01 

4757.66 

.000 

.978 

Factor A (training group vs. 
control group 1 vs. control 
group 2) 

MCSL 

156.25 

2 

78.13 

7.36 

.001 ** 

.122 

MCSR 

113.62 

2 

56.81 

3.92 

.023 * 

.069 


FAS 

131.52 

2 

65.76 

6.30 

.003 ** 

.106 


TUT 

199.15 

2 

99.58 

7.43 

.001 ** 

.123 

Factor B (pre- and post-test 
vs. post-test only) 

MCSL 

7.24 

1 

7.24 

.68 

.411 

.006 

MCSR 

34.55 

1 

34.55 

2.38 

.126 

.022 


FAS 

16.73 

1 

16.73 

1.60 

.208 

.015 


TUT 

20.20 

1 

20.20 

1.52 

.222 

.014 

Interaction 
(Factor A * Factor B) 

MCSL 

146.09 

2 

73.04 

6.88 

.002 ** 

.115 

MCSR 

36.58 

2 

18.29 

1.26 

.288 

.023 


FAS 

4.26 

2 

2.13 

.20 

.816 

.004 


TUT 

75.64 

2 

37.82 

2.82 

.064 

.051 

Error 

MCSL 

1125.86 

106 

10.62 





MCSR 

1537.82 

106 

14.51 





FAS 

1107.34 

106 

10.45 





TUT 

1421.32 

106 

13.41 




Total variation 

MCSL 

71631.40 

112 






MCSR 

85877.03 

112 






FAS 

81396.83 

112 






TUT 

67342.67 

112 





Adjusted total variation 

MCSL 

1431.26 

111 
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MCSR 

FAS 

TUT 

a. R 2 = .213 (adjusted R 2 = .176) 

b. R 2 = .095 (adjusted R 2 = .052) 

c. R 2 = .117 (adjusted R 2 = .075) 

d. R 2 = ,175 (adjusted R 2 = ,136) 


1698.72 111 


1253.83 111 


1721.94 111 


* F value statistically significant (p < .05, two-tailed) ** F value statistically significant {p <.01, two-tailed) 


The effect size for the training group between pre- and post-test was largest for metacognitive 
skills of the tutor related to guiding learning groups (MCSL). Smaller effects could be 
observed for metacognitive skills of the tutor related to self-regulation (MCSR), facilitator 
skills (FAS), and tutor skills (TUT) (Table 7). 

Table 7 

Effect sizes (Cohen’s d) for the training group with pre- and post-test (n=21) 



M 

pre¬ 

test 

M 

post¬ 

test 

SD 

pre¬ 

test 

SD 

post¬ 

test 

Cohen's 

d 

MCSL 

23.7 

28.4 

5.67 

5.76 

0.84 

MCSR 

27.3 

28.8 

6.86 

5.66 

0.24 

FAS 

25.7 

27.9 

10.50 

9.61 

0.22 

TUT 

23.9 

26.8 

8.18 

13.69 

0.27 


MCSL: Metacognitive skills related to guiding 
learning groups, MCSR: Metacognitive skills related 
to self-regulation, FAS: Facilitator skills, TUT: Tutor 
skills 


CONCLUSIONS 

The data indicate and support the effectiveness of the training programme for pbl tutors for 
developing metacognitive skills related to guiding and steering learning groups in a pbl 
tutorial. However, there were only small effects for the development of facilitator skills and 
tutor skills. We conclude from our date that the training should include more exercises for 
building these skills in the future. It also needs to be considered that the newly trained and 
certified PBL tutors have completed the questionnaire right at the end of the training 
programme. Many of them have had no or very limited experience with facilitating tutorials 
outside the training programme. In follow-up measures we need to evaluate the mid-term and 
long-term effects of the training programme on tutor effectiveness. 
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There are strong correlations between self-reported metacognitive skills related to guiding 
learning groups and self-regulation on the one side, and both facilitator and tutor skills on the 
other side. This supports the conclusion that a PBL tutor training programme should not only 
cover the technical aspects of problem based learning (e. g. instructing, stimulating, probing 
questions, elaborating) but also support the development of reasoning and reflection skills as 
described in the 3C3R framework of Hung (2006). 

Our study demonstrates the added value of intensive training for prospective PBL tutors 
compared with other methods, e.g. self-study of a PBL tutor guide only without 
complementary training, coaching or advice. This does not conclude that the available tutor 
guides are not helpful or supportive. However, self-study of these training materials might not 
be enough to develop the critical metacognitive and behavioural skills in order to achieve best 
performance as a PBL tutor. 


SUMMARY, DISCUSSION AND OUTLOOK 

Overall, the “Train the Tutor” Programme has shown satisfactory effects on the development 
of metacognitive skills related to guiding learning groups (Effect size (Cohen’s d) = .84). The 
effects for other dependent variables (MCSR; FAS, TUT) was still measurable, but smaller 
(.Effect sizes (Cohen’s d) = .22 - .27). In order to reach a stronger effect size for 
metacognitive-skills related to self-regulation and facilitator skills the training needs to be 
modified and should include more specific exercises for developing these competencies in 
particular. 

For example, the training participants could be challenged more with difficult group situations 
(e. g. low participation, active or passive resistance of the group members to tutor 
interventions), in which they need to reflect first how these negative stimuli affect their self¬ 
regulation (cognitive, emotion, motivation) and then choose and execute appropriate 
interventions. This conclusion is supported by the low score on the item “I have no problems 
to deal effectively with “difficult” participants in a group setting (e. g. very dominating 
people).” (Mean = 3.62; SD = .91) (ANNEX 1). In comparison, the overall self-assessment 
after the training through the participants was higher on facilitation skills (Mean = 26.81; SD 
= 3.55) rather than tutor skills (Mean = 24.24; SD = 4.03) (ANNEX 1). This indicates that in 
the next run the training needs to be adjusted in a way that intensifies PBL tutor skills as 
described in the Tutor Intervention Profile (TIP). 

In addition, the prospective tutors should be trained better how to construct appropriate and 
challenging problems for themselves before presenting problems to others. This is concluded 
from the comparably low score and part-whole correlation of the item “I find it easy to design 


53 



M. Miihlfelder, T. Konermann, L.-M. Borchard 


JPBLHE: VOL. 3, No. 2, 2015 


PBL cases for students to share, discuss, and learn.” (Mean = 3.07; SD = .98; r tt = .27) 
(ANNEX 1). One way of doing this is the opportunity to assign the training participants to 
define and describe task-related problems and let them work through the process. Later they 
should reflect their learning process and report back to others about their observations and key 
learning points. More than that, the empirical data supports the importance of active learning 
and group based learning for an effective PBL “Train the Tutor” (TtT) process compared to 
self-study (control group 1) or no training at all (control group 2). 

The pre-test vs. pre-/post-test condition had no effect on the post-test results for self-reported 
tutor skills; except, there was a strong interaction effect between the factors A (training group 
vs. control groups) and B (pre-/post-test vs. post-test only) for the dependent variable MCSL. 
The subjects who had completed the pre-test before and took part in the training had the 
highest scores on this scale. This indicates the possibility that the awareness of the items in 
the pre-test has focussed and primed the training participants with pre-test experience more 
than those in the post-test condition only. 

Metacognitive skills of PBL tutors were positively correlated with student satisfaction 
measures for the learning outcome, not so much with the learning content or the learning 
process. Lacilitator skills were positively correlated with both student satisfaction with the 
learning outcome and content. 

More experimental and better controlled studies should investigate the cognitive, affective 
and behavioural mechanisms of effective PBL tutorials in detail. Especially the quality of the 
relationship between the tutor and the students might be relevant for both student satisfaction 
and the learning outcome. This conclusion is supported by other research results which 
describe that the development of effective tutor behaviour is an effective way to improve the 
learning process and achievement of the students in a PBL curriculum (Chng, Yew & 
Schmidt, 2011; Schmidt & Moust, 2000, Wetzel, 1996). 

Another way for elaborating this study further could be to videotape the interaction between 
tutor and students and to interview the tutor later while showing him/her the video. He/she 
then might verbalize his observations, intentions and metacognitive strategies during the 
various phases of the tutorial. 

Problem based learning has a lot of potential for improving the learning effectiveness of self- 
regulated learning groups in secondary and higher education (Azer, 2008; Weber, 2004). Well 
trained and capable tutors play a crucial role in this setting. The more we want to shift from 
teaching to learning in the curricula the higher becomes the importance of creating supporting 
organizational structures for learning and development. Developing and training a sufficient 
number of effective PBL tutors is one critical element of such a learning architecture. 
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LIMITATIONS 

The data gathered in this study are limited in terms of reliability. While the questionnaire for 
self-reported tutor skills shows acceptable though not excellent values of internal consistency 
(Cronbach’s a 0.69 - 0.78), the reliability of the TIP data can be challenged due to the limited 
number of observations and observers (n=14). There is also a lack of qualitative data, e.g. 
from interviews with participants before, during, and after completion of the training 
programme. In an improved “mixed methods” design, the combination of qualitative with 
quantitative data should be pre-considered in order to cross-validate the data. Due to these 
limitations it cannot be clarified definitely how large the effect size of the training programme 
on the dependent variables really was, and to which extend other factors like maturation over 
time or the self-selection of training participants have influenced the observed behaviours of 
the PBL tutors and their effectiveness. 

The strong inter-correlations of the four scales of the questionnaire (MCSL, MCSR, FAS, 
TUT) indicate a strong common factor underlying the data structure. A confirmatory factor 
analysis of the data has shown a rather inconsistent image. More research is needed to 
increase the psychometric quality of the questionnaire applied in this exploratory study. 

The non-randomized allocation of participants to the training group has limited both the 
internal validity and the generalizability of our conclusions. Therefor it is necessary to repeat 
this study in a more controlled experimental setting with completely randomized groups in 
order to test potential effects of selection or self-selection of training participants on the 
results. 
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ANNEX 1 

Questionnaire for measuring self-reported meta-cognitive skills, facilitator skills, and tutor skills 


1) MCSL (Meta-cognitive skills related to guiding learning groups) 


Item 

No. 

Item 

Mean 

SD 

r tt 

Cronbach’s a 
if item deleted 

1 

I have a large variety of behavioural strategies 
how to steer group dynamics. 

3.50 

.78 

.47 

.67 

2 

I know how to deal with difficult situations in 
group settings (e, g, interpersonal conflicts). 

3.70 

.77 

.41 

.69 

3 

Before engaging in a group situation I have plan 
what to do in order to reach the (learning) goals of 
the tutorial. 

3.53 

.85 

.59 

.64 

4 

I can judge in advance how a tutorial group will 
react to my questions, guidance, and interventions. 

(*) 

3.47 

.86 

.27 

.72 

5 

I can easily reflect and understand the reasons 
when a tutorial group is not collaborating 
effectively. 

3.85 

.90 

.44 

.67 

6 

I find it easy to integrate different types of people 
in the collaborative learning process. 

3.53 

.98 

.44 

.68 

7 

I know how to evaluate the effectiveness of the 
collaborative learning process in a tutorial. 

3.51 

.88 

.37 

.70 


(*) item deleted due to unsatisfactory r tt and higher Cronbach’s a if item deleted 
Scale statistics 


Sample size 

Number of 

Mean: 

Min: 

Max: 

SD: 

Cronbach’s a for 

(valid cases): 
n = 90 

items: 7 

25.11 

17 

34 

3.66 

standardized items: 
.72 
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2) MCSR (Meta-cognitive skills related to self-regulation) 


Item 

No. 

Item 

Mean 

SD 

r„ 

Cronbach’s a 
if item deleted 

1 

I can easily judge alternatives for my actions at any 
time. 

3.80 

.74 

.53 

.69 

2 

Especially in stressful situations I can verbalize my 
feelings and emotions very well. 

3.69 

1.05 

.39 

.71 

3 

I have a clear mental model of how to plan, do, and 
check my actions and their behavioural and 
emotional effects. 

3.85 

.87 

.33 

.72 

4 

I am able to think through various alternatives for 
action paths and evaluate their consequences. 

3.79 

.90 

.49 

.69 

5 

I am aware of my emotions when doing things. 

4.14 

.98 

.51 

.68 

6 

I reflect my actions regularly and ask others for 
feedback. 

3.78 

.96 

.39 

.71 

7 

If I do not know the answer to a problem I am able 
to admit that, and I know whom to address to for 
support. 

4.34 

.86 

.50 

.69 


Scale statistics 


Sample size 

Number of 

Mean: 

Min: 

Max: 

SD: 

Cronbach’s a for 

(valid cases): 
n = 94 

items: 7 

27.39 

16 

35 

3.95 

standardized 
items: .74 
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3) FAS (Facilitator skills) 


Item 

No. 

Item 

Mean 

SD 

Ft 

Cronbach’s a 
if item deleted 

1 

I am good at active listening. 

4.38 

.82 

.53 

.61 

2 

I can easily structure group discussions. 

3.65 

.82 

.38 

.65 

3 

I am able to summarize the results of group 
discussions. 

3.84 

.88 

.43 

.64 

4 

I am able to visualise ideas and concepts on a white 
board or flip chart. 

3.68 

.97 

.32 

.67 

5 

I have no problems to deal effectively with 
“difficult” participants in a group setting (e. g. very 
dominating people). 

3.62 

.91 

.40 

.65 

6 

I am able to manage and keep the time in group 
settings. 

3.66 

.83 

.31 

.67 

7 

I keep friendly and treat everyone respectfully, 
especially in difficult group situations. 

3.98 

.80 

.39 

.65 


Scale statistics 


Sample size 

Number of 

Mean: 

Min: 

Max: 

SD: 

Cronbach’s a for 

(valid cases): 
n = 92 

items: 7 

26.81 

15 

34 

3.55 

standardized 
items: .69 
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4) TUT (Tutor skills) 


Item 

No. 

Item 

Mean 

SD 

r t t 

Cronbach’s a 
if item deleted 

1 

I find it easy to design PBL cases for students to 
share, discuss, and learn. (*) 

3.07 

.98 

.27 

.80 (*) 

2 

I can easily evaluate different levels of knowledge 
and subject matter understanding of students in a 
tutorial group. 

3.34 

.85 

.54 

.74 

3 

I can easily integrate people with different learning 
skills in a learning group. 

3.37 

.86 

.50 

.75 

4 

I have always an idea how I can support a learning 
group that is struggling with a task. 

3.16 

.85 

.62 

.73 

5 

I am able to create a positive atmosphere and 
learning climate in a group. 

3.74 

.93 

.64 

.72 

6 

I am able to stimulate interaction and individual 
accountability in a learning group. 

3.57 

.78 

.54 

.74 

7 

I find it easy to provide feedback to a group 
regarding the effectiveness of their learning process. 

3.98 

.90 

.45 

.76 


(*) item deleted due to unsatisfactory r„ and higher Cronbach’s a if item deleted 
Scale statistics 


Sample size 

Number of 

Mean: 

Min: 

Max: 

SD: 

Cronbach’s a for 

(valid cases): 
n = 94 

items: 7 

24.24 

12 

33 

4.03 

standardized 
items: .78 
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ANNEX 2 

Questionnaire for measuring the satisfaction of students at the end of the tutorial 


1) SLO (Satisfaction with the learning outcome) 


Item 

No. 


0 

1 

2 

3 

4 



Item 

very 

dissatisfied 

dissatisfied 

neutral 

satisfied 

very 

satisfied 

Mean 

SD 

1 

Overall, I am 
satisfied with the 
learning outcome 
of this tutorial. 






3.0 

0.6 


2) SLP (Satisfaction with the learning process) 


Item 

No. 


0 

1 

2 

3 

4 



Item 

very 

dissatisfied 

dissatisfied 

neutral 

satisfied 

very 

satisfied 

Mean 

SD 

1 

Overall, I am 
satisfied with the 
learning process 
of this tutorial 
regarding pace 
and structure. 






3.1 

0.7 


3) SLC (Satisfaction with the learning content) 


Item 

No. 

Item 

0 

very 

dissatisfied 

1 

dissatisfied 

2 

neutral 

3 

satisfied 

4 

very 

satisfied 

Mean 

SD 

1 

Overall, I am 
satisfied with the 
learning content 
of this tutorial. 






2.8 

0.4 


61 






